System and method for providing search query refinements

ABSTRACT

A system and method for providing search query refinements are presented. A stored query and a stored document are associated as a logical pairing. A weight is assigned to the logical pairing. The search query is issued and a set of search documents is produced. At least one search document is matched to at least one stored document. The stored query and the assigned weight associated with the matching at least one stored document are retrieved. At least one cluster is formed based on the stored query and the assigned weight associated with the matching at least one stored document. The stored query associated with the matching at least one stored document are scored for the at least one cluster relative to at least one other cluster. At least one such scored search query is suggested as a set of query refinements.

CROSS-REFERENCE TO RELATED APPLICATION

This non-provisional patent application claims priority under 35 USC §119(e) to U.S. provisional patent application Ser. No. 60/500,539, filedSep. 5, 2003, the disclosure of which is incorporated by reference.

FIELD OF THE INVENTION

The present invention relates in general to query processing and, inparticular, to a system and method for providing search queryrefinements.

BACKGROUND OF THE INVENTION

Although the Internet traces back to the late 1960s, the widespreadavailability and acceptance of personal computing and internetworkinghave resulted in the explosive growth and unprecedented advances ininformation sharing technologies. In particular, the Worldwide Web(“Web”) has revolutionized accessibility to untold volumes ofinformation in stored electronic form to a worldwide audience, includingwritten, spoken (audio) and visual (imagery and video) information, bothin archived and real-time formats. In short, the Web has provideddesktop access to every connected user to a virtually unlimited libraryof information in almost every language worldwide.

Search engines have evolved in tempo with the increased usage of the Webto enable users to find and retrieve relevant Web content in anefficient and timely manner. As the amount and types of Web content haveincreased, the sophistication and accuracy of search engines havelikewise improved. Generally, search engines strive to provide thehighest quality results in response to a search query. However,determining quality is difficult, as the relevance of retrieved Webcontent is inherently subjective and dependent upon the interests,knowledge and attitudes of the user.

Existing methods used by search engines are based on matching searchquery terms to terms indexed from Web pages. More advanced methodsdetermine the importance of retrieved Web content using, for example, ahyperlink structure-based analysis, such as described in S. Brin and L.Page, “The Anatomy of a Large-Scale Hypertextual Search Engine,” (1998)and in U.S. Pat. No. 6,285,999, issued Sep. 4, 2001 to Page, thedisclosures of which are incorporated by reference.

A typical search query scenario begins with either a natural languagequestion or individual terms, often in the form of keywords, beingsubmitted to a search engine. The search engine executes a searchagainst a data repository describing information characteristics ofpotentially retrievable Web content and identifies the candidate Webpages. Searches can often return thousands or even millions of results,so most search engines typically rank or score only a subset of the mostpromising results. The top Web pages are then presented to the user,usually in the form of Web content titles, hyperlinks, and otherdescriptive information, such as snippets of text taken from the Webpages.

Providing quality search results can be complicated by the literal andimplicit scope of the search query itself. A poorly-framed search querycould be ambiguous or be too general or specific to yield responsive andhigh quality search results. For instance, terms within a search querycan be ambiguous at a syntactic or semantic level. A syntactic ambiguitycan be the result of an inadvertent homonym, which specifies anincorrect word having the same sound and possibly same spelling, butdifferent meaning from the word actually meant. For example, the word“bear” can mean to carry or can refer to an animal or an absence ofclothing. A semantic ambiguity can be the result of improper context.For example, the word “jaguar” can refer to an animal, a version of theMacintosh operating system, or a brand of automobile. Similarly, searchterms that are too general result in overly broad search results whilesearch terms that are too narrow result in unduly restrictive andnon-responsive search results.

Accordingly, there is a need for an approach to providing suggestionsfor search query refinements that will resolve ambiguities or overgeneralities or over specificities occurring in properly framed searchqueries. Preferably, such an approach would provide refined searchqueries that, when issued, result in search results closely related tothe actual topic underlying the intent of the original search query andprovide suggestions that reflect conceptual independence and clearmeanings as potential search terms.

SUMMARY OF THE INVENTION

An embodiment provides a system and method for creating query refinementsuggestions. At least one search document retrieved responsive to aquery is matched to one or more stored queries. The stored query isscored as a potential query refinement suggestion.

A further embodiment provides a system and method for providing searchquery refinements. A stored query and a stored document are associatedas a logical pairing. A weight is assigned to the logical pairing. Thesearch query is issued and a set of search documents is produced. Atleast one search document is matched to at least one stored document.The stored query and the assigned weight associated with the matching atleast one stored document are retrieved. At least one cluster is formedbased on the stored query and the assigned weight associated with thematching at least one stored document. The stored query associated withthe matching at least one stored document are scored for the at leastone cluster relative to at least one other cluster. At least one suchscored search query is suggested as a set of query refinements.

A further embodiment provides a system and method for integrating queryrefinement candidates. At least one search document retrieved responsiveto a query is matched to one or more stored documents associated with astored query and weight. At least one cluster is formed based on thestored query and weight associated with each stored document matchedresponsive to the query. At least one further search document retrievedresponsive to a candidate query is matched to the one or more storeddocuments. At least one further cluster is formed based on the storedquery and weight associated with each stored document matched responsiveto the candidate query. The at least one cluster and the at least onefurther cluster are combined. The stored query for the combined clusterrelative to at least one other cluster is scored as a potential queryrefinement suggestion.

Still other embodiments of the present invention will become readilyapparent to those skilled in the art from the following detaileddescription, wherein are described embodiments of the invention by wayof illustrating the best mode contemplated for carrying out theinvention. As will be realized, the invention is capable of other anddifferent embodiments and its several details are capable ofmodifications in various obvious respects, all without departing fromthe spirit and the scope of the present invention. Accordingly, thedrawings and detailed description are to be regarded as illustrative innature and not as restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a system for providing search queryrefinements, in accordance with the present invention.

FIG. 2 is a functional block diagram showing a precomputation server, inaccordance with the present invention.

FIG. 3 is a functional block diagram showing a query refinement server,in accordance with the present invention.

FIG. 4 is a flow diagram showing a method for providing search queryrefinements, in accordance with the present invention.

FIG. 5 is a flow diagram showing the routine for performingprecomputation for use in the method of FIG. 4.

FIGS. 6-9 are flow diagrams showing the routines for referencing querysources for use in the method of FIG. 5.

FIG. 10 is a flow diagram showing the routine for performing on-linequery refinements for use in the method of FIG. 4.

FIG. 11 is a flow diagram showing a routine for integrating candidatequery refinements for use in the method of FIG. 4, in accordance with afurther embodiment.

DETAILED DESCRIPTION

System Overview

FIG. 1 is a block diagram showing a system 10 for providing search queryrefinements, in accordance with the present invention. A plurality ofindividual clients 12 are communicatively interfaced to a server 11 viaan internetwork 13, such as the Internet, or other form ofcommunications network, as would be recognized by one skilled in theart. The individual clients 12 are operated by users 19 who transactrequests for Web content and other operations through their respectiveclient 12.

In general, each client 12 can be any form of computing platformconnectable to a network, such as the internetwork 13, and capable ofinteracting with application programs. Exemplary examples of individualclients include, without limitation, personal computers, digitalassistances, “smart” cellular telephones and pagers, lightweightclients, workstations, “dumb” terminals interfaced to an applicationserver, and various arrangements and configurations thereof, as would berecognized by one skilled in the art. The internetwork 13 includesvarious topologies, configurations, and arrangements of networkinterconnectivity components arranged to interoperatively couple withenterprise, wide area and local area networks and include, withoutlimitation, conventionally wired, wireless, satellite, optical, andequivalent network technologies, as would be recognized by one skilledin the art.

For Web content exchange and, in particular, to transact searches, eachclient 12 executes a Web browser 18 (“Web browser”), which preferablyimplements a graphical user interface and through which search queriesare sent to a Web server 20 executing on the server 11. Each searchquery describes or identifies information, generally in the form of Webcontent, which is potentially retrievable via the Web server 20. Thesearch query provides search characteristics, typically expressed asindividual terms, such as keywords and the like, and attributes, such aslanguage, character encoding and so forth, which enables a search engine21, also executing on the server 11, to identify and send back searchresult documents, generally in the form of Web pages. Other styles,forms or definitions of search queries and characteristics are feasible,as would be recognized by one skilled in the art.

The Web pages are sent back to the Web browser 18 for presentation,usually in the form of Web content titles, hyperlinks, and otherdescriptive information, such as snippets of text taken from the Webpages. The user can view or access the Web pages on the graphical userinterface and can input selections and responses in the form of typedtext, clicks, or both. The server 11 maintains a search database 15 inwhich Web content 22 is maintained. The Web content 22 could also bemaintained remotely on other Web servers (not shown) interconnectedeither directly or indirectly via the internetwork 13 and which arepreferably accessible by each client 12. In a further embodiment, theserver 11 maintains a cache 23 in which cached documents 24 and cachedqueries 25 are maintained. The cache 23 associates each cached document24 with one or more cached queries 25 to improve searching performance,as is known in the art. Finally, in a still further embodiment, thesearch engine 21 maintains a query log 26 in which records of previoussearch queries 27 are tracked.

The search engine 21 preferably identifies the Web content 22 bestmatching the search characteristics to provide high quality Web pages,such as described in S. Brin and L. Page, “The Anatomy of a Large-ScaleHypertextual Search Engine” (1998) and in U.S. Pat. No. 6,285,999,issued Sep. 4, 2001 to Page, the disclosures of which are incorporatedby reference. In identifying matching Web content 22, the search engine21 operates on information characteristics describing potentiallyretrievable Web content. Note the functionality provided by the server20, including the Web server 20 and search engine 21, could be providedby a loosely- or tightly-coupled distributed or parallelized computingconfiguration, in addition to a uniprocessing environment.

Search queries can potentially be ambiguous or lack generality orspecificity. Such poorly-framed search queries can be remedied throughsearch query refinements, which can be provided in response to searchquery issuances. Search query refinements are generated and suggested asa two-part operation. First, a search query is associated with a searchresult document in a one-to-one association and a weight is assigned toeach search query-search result document association, as furtherdescribed below with reference to FIG. 2. Second, the searchquery-search result document associations and assigned weights arematched to at least one new search query to form scored clusters, asfurther described below with reference to FIG. 3. The clusters areprojected from multi-dimensional space based on constituent query termsand unique search queries associated with any document in each clusterare scored. The clusters are named based on the scoring, preferably byhighest scores. The named clusters are sorted and provided as suggestedrefinements to the original search query.

The individual computer systems, including server 11 and clients 12,include general purpose, programmed digital computing devices consistingof a central processing unit (processors 13 and 16, respectively),random access memory (memories 14 and 17, respectively), non-volatilesecondary storage, such as a hard drive or CD ROM drive, network orwireless interfaces, and peripheral devices, including user interfacingmeans, such as a keyboard and display. Program code, including softwareprograms, and data is loaded into the RAM for execution and processingby the CPU and results are generated for display, output, transmittal,or storage. The Web browser 18 is an HTTP-compatible Web browser, suchas the Internet Explorer, licensed by Microsoft Corporation, Redmond,Wash.; Navigator, licensed by Netscape Corporation, Mountain View,Calif.; or other forms of Web browsers, as are known in the art.

Precomputation Server

FIG. 2 is a functional block diagram 30 showing a precomputation server34, in accordance with the present invention. The precomputation system31 builds a set of associated queries and documents, preferably as anoff-line operation. The precomputation system 31 includes aprecomputation engine 34, which associates a stored query 40 with astored document 41 and assigns a weight 43 to each association 42, asfurther described below with reference to FIG. 5.

The precomputation system 31 builds and maintains the associationdatabase 39. The association database 39 stores the stored queries 40,stored documents 41, associations 42, and weights 43, which are used bya query refinement server to formulate and suggest query refinements, asfurther described below with reference to FIG. 3. In furtherembodiments, the precomputation system 31 also references the query log26, which is stored in the search database 15, and the cached documents24 and cached queries 25, which are stored in the cache (both shown inFIG. 1).

The precomputation engine 34 logically includes four modules. Otherlogical arrangements and definitions of functional modules are possible,as would be recognized by one skilled in the art. First, an associator35 associates a stored query 40 with a stored document 41 and the storedquery-stored document pairing is maintained in the association database39 as a set of associations 42. Each stored query 40 is associated withonly one stored document 41, although any given stored query 40 can bepaired with one or more stored documents 41 in a one-to-manyrelationship. Each individual pairing is maintained as a separateassociation 42 in the association database 39. However, the individualpairings need not be explicitly stored in the form of associations 42and can instead be logically recorded or tracked, such as by using amapping, table or other means for matching stored queries 40 with storeddocuments 41, as would be recognized by one skilled in the art.

The associator 35 also assigns a weight 43 to each association 42. Eachweight 43 estimates the relevance of the stored document 41 to theassociated stored query 40. When multiple associations 42 for the samestored query-stored document pairing occurs, the weights 43 for each ofthe multiple associations 42 are summed. If query frequency data isavailable, such as, for example, when a search query 59 is cached, theweight 43 is multiplied by the search query frequency, that is, thenumber of times that the search query 59 has been issued.

Second, a selector 36 selects one or more of the stored documents 41 forassociation with a stored query 40 based on an issued search. In thedescribed embodiment, the selector 36 selects the stored documents 41,which are each separately associated with a stored query 40 based oneither a search document chosen following an issued search or from a setof search results received for an issued search, as further describedbelow respectively with reference to FIGS. 6 and 7.

Third, the regenerator 37 selects one or more of the stored documents41, which are each separately associated with a stored query 40 based onthe query log 26. In a further embodiment, the regenerator 37regenerates the search results from previously tracked queries 27, asreflected in the query log 26. The regenerator 37 selects theregenerated search results as stored documents 41, which are eachseparately associated with a previously tracked search query 27, asfurther described below with reference to FIG. 8.

Fourth, the inverter 38 selects one or more of the stored documents,which are each separately associated with a stored query 40 based oncached data. In a still further embodiment, the inverter 38 evaluatesthe cached documents 24 and cached queries 25 and inverts the cacheddocument-cached queries pairings into cached query-cached documentspairings. The inverter 38 selects the inverted cached documents 24 asstored documents 41, which are each separately associated with a cachedquery 25, as further described below with reference to FIG. 9.

The individual computer system, including the precomputation system 31,include general purpose, programmed digital computing devices consistingof a central processing unit (processor 33), random access memory(memory 32), non-volatile secondary storage, such as a hard drive or CDROM drive, network or wireless interfaces, and peripheral devices,including user interfacing means, such as a keyboard and display.Program code, including software programs, and data is loaded into theRAM for execution and processing by the CPU and results are generatedfor display, output, transmittal, or storage. Note the functionalityprovided by the precomputation system 31 could be provided by a loosely-or tightly-coupled distributed or parallelized computing configuration,in addition to a uniprocessing environment.

Query Refinement Server

FIG. 3 is a functional block diagram 50 showing a query refinementsystem 51, in accordance with the present invention. The queryrefinement system 51 formulates and suggests one or more queryrefinements 67. The query refinements can be formulated either as anon-line operation following a search query issuance or based onprecomputations for a given set of search queries. The query refinementsystem 51 includes a query refinement engine 54, which formulates thequery refinements 67 as suggestions in response to an actual searchquery 59, as further described below with reference to FIG. 10, andwhich integrates candidate query refinements 68, as further describedbelow with reference to FIG. 11.

The query refinement engine 54 logically includes four modules. Otherlogical arrangements and definitions of functional modules are possible,as would be recognized by one skilled in the art. First, a matcher 55matches one or more of the stored documents 41 to the actual searchdocuments 60, generated by the search engine 21 (shown in FIG. 1) inresponse to the issuance of a search query 59. The search engine 21 alsogenerates relevance scores 60 as part of the search query issuance. Thematcher 55 identifies the stored queries 40 and assigned weights 43using the associations 43 corresponding to the matched stored documents41.

Second, a clusterer 56 forms one or more clusters 62 based on termvectors 62 formed from the terms occurring in the matched stored queries40 and corresponding weights 43. The term vectors 62 are normalizedvectors projected into multi-dimensional space, with each dimensioncorresponding to a term, which can be an individual word or wordcombination. The clusters 62 are ranked based on the relevance scores 61assigned to the search documents 60 corresponding to the matched storeddocuments 41 and the number of stored documents 41 occurring in eachcluster 62. The highest ranking clusters 62 are selected as potentialrefinement clusters 64. In the described embodiment, the clusters 62 areformed using a hierarchical agglomerative clustering algorithm, such asdescribed in E. Rasmussen, “Clustering Algorithms,” in “InformationRetrieval,” (W. Frakes & R. Baeza-Yates eds. 1992), the disclosure ofwhich is incorporated by reference, although other types of clusteringalgorithms could be used, as would be recognized by one skilled in theart.

Third, a scorer 57 computes center-weighted term vectors, referred to ascentroids 65, which each represent the weighted center of the termvector 62 of each cluster 63. The centroids 65 are computed from each ofthe potential refinement clusters 64. The scorer 57 assigns scores 66 toeach unique search query in each of the potential refinement clusters 64based on the number of stored documents 41 with which the search queryis associated and the distance from the centroid 65. Other approaches tocomputing centroids 65 could also be used, including using unweightedvalues and by varying the forms of weighting and averaging, as would berecognized by one skilled in the art.

Finally, a presenter 58 identifies the substantially highest scoringsearch queries as one or more query refinements 67 to the user 19.Importantly, the details of the formulation of the search queryrefinements, including the term vectors 62, clusters 63, potentialrefinement clusters 64, centroids 65, and scores 66, are encapsulated bythe query refinement engine 54. Thus, a user 19 will only be aware ofthe actual suggested query refinements 67.

The individual computer system, including the query refinement system51, include general purpose, programmed digital computing devicesconsisting of a central processing unit (processor 53), random accessmemory (memory 52), non-volatile secondary storage, such as a hard driveor CD ROM drive, network or wireless interfaces, and peripheral devices,including user interfacing means, such as a keyboard and display.Program code, including software programs, and data is loaded into theRAM for execution and processing by the CPU and results are generatedfor display, output, transmittal, or storage. Note the functionalityprovided by the query refinement system 51 could be provided by aloosely- or tightly-coupled distributed or parallelized computingconfiguration, in addition to a uniprocessing environment.

Method Overview

FIG. 4 is a flow diagram showing a method 70 for providing search queryrefinements, in accordance with the present invention. The method 70 isdescribed as a sequence of process operations or steps, which can beexecuted, for instance, by the precomputation engine 34 of FIG. 2 andthe query refinement engine 54 of FIG. 3, or equivalent components.

The method 70 performs precomputation (Block 71) and query refinement(Block 72). Precomputation (Block 71) builds the association database 39(shown in FIG. 2) by associating and storing representative storedqueries 40 with stored documents 41, along with associations 42 andweights 43, as further described below with reference to FIG. 5. In thedescribed embodiment, precomputation is performed as an off-lineoperation independent of any specific search query.

Query refinement (Block 72) formulates query refinement suggestionsstructured to better frame search queries posted by users. In oneembodiment, the query refinement suggestions are performed as an on-lineoperation based on search query results observed for issued searchqueries, which can be matched and applied to the data maintained in theassociation database 39, as further described below with reference toFIG. 10. In a further embodiment, candidate query refinements 68 can beintegrated into query refinement suggestions, which can also be matchedand applied to the data maintained in the association database 39, asfurther described below with reference to FIG. 11.

The method terminates upon the completion of the last query refinementoperation.

Precomputation Routine

FIG. 5 is a flow diagram showing the routine 80 for performingprecomputation for use in the method 70 of FIG. 4. The purpose of thisroutine is to build the association database 39 (shown in FIG. 3) andassociate stored queries 40 with stored documents 41 to formassociations 42 and assign weights 43.

Initially, query sources are referenced to build the stored queries 40and stored documents 41 maintained in the association database 39.Briefly, query source referencing refers to identifying appropriatepairings of search queries and search result documents based on actualsearch query issuances, including current, stored and cached searchquery issuances. Thus, the stored queries 40 can originate from actualsearch queries, as further described below with reference to FIGS. 6 and7, previous search queries 27 tracked in the query log 26, as furtherdescribed below with reference to FIG. 8, and cached queries 25maintained in the cache 13, as further described below with reference toFIG. 9. In addition, the stored documents 41 can be search documents 60retrieved in response to actual search queries 59, or can be regeneratedsearch results or cached documents 24 maintained in the cache 13. Othersources of search queries and documents for association as storedqueries 40 and stored documents 41 are possible, as would be recognizedby one skilled in the art.

Once the stored queries 40 for the association database 39 have beenidentified and stored, each stored query 40 is iteratively processed(Block 82), as follows. A stored query 40 is separately associated witheach stored document 41 (Block 83). A one-to-one association 42 isformed, provided, however, that each associated stored document 41 isbased on the issuance of the corresponding associated stored query 40.Each stored query 40 is separately associated with only one storeddocument 41, although any given stored query 40 can be paired with oneor more stored documents 41 in a one-to-many relationship. A weight 43is assigned to the association 42 (Block 84), reflecting the relevanceof the stored document 41 to the stored query 40. Processing continueswith each remaining stored query 40 (Block 85), after which the routinereturns.

Query Source Referencing Routines

FIGS. 6-9 are flow diagrams showing the routines for referencing querysources for use in the method 80 of FIG. 5. The purposes of theseroutines are to populate the stored queries 40 and stored documents 41maintained in the association database 39. Each of the routinesidentifies search queries 59 and related search documents 60,respectively for use as stored queries 40 and stored documents 41.

Each stored query 40 in an association 42 is associated with a storeddocument 41, although any given stored query 40 can be paired with oneor more stored documents 41 in a one-to-many relationship. Eachindividual pairing is maintained as a separate association 42. However,the individual pairings need not be explicitly stored in the form ofassociations 42 and can be logically recorded or tracked, such as byusing a mapping, table or other means for matching stored queries 40with stored documents 41, as would be recognized by one skilled in theart.

In the association database, each of the stored queries 40 take the formof a search query 59 expressed as, for instance, keywords or terms.Terms include individual words or combinations of words. Each of thestored documents 41 are preferably stored as references and not asactual content. Generally, each stored document 41 can be in the formof, by way of example, a uniform resource locator (URL), hyperlink,anchor, or document excerpt.

Query Source Referencing Using a Chosen Search Document

FIG. 6 is a flow diagram showing a routine 90 for performingprecomputation based on a chosen search document. The purpose of thisroutine is to identify a one-to-one pairing relationship between asearch query 59 and a selected search document 60. The one-to-onerelationship can occur, for instance, when a user 19 selects aparticular search document 60 from among a set of search resultsfollowing the issuance of a search query 59.

First, a search query 59 is issued (Block 91) and search results,consisting of search documents 60 (shown in FIG. 3), are received (Block92). The actual search query 59 is selected as the stored query 40(Block 93). A search document 59 is selected as a stored document 41,when the search document 59 is chosen by the user 19 performing thesearch from among the search results (Block 94). The routine thenreturns.

Query Source Referencing Using Search Results

FIG. 7 is a flow diagram showing a routine 100 for performingprecomputation based on a search results. The purpose of this routine isto identify a one-to-many pairing relationship between a search query 59and a set of search results 60. The one-to-many relationship occurs whena set of search results are identified following the issuance of asearch query 59.

The search is query issued (Block 101) and search results, consisting ofsearch documents 60 (shown in FIG. 3), are received (Block 102). Theactual search query 59 is selected as the stored query 40 (Block 103).The set of search documents 59 are selected as stored documents 41(Block 104). Note that each search query is separately associated withonly one search document and multiple search query-search documentassociations can be formed for any given search query. The routine thenreturns.

Query Source Referencing Using Tracked Queries

FIG. 8 is a flow diagram showing a routine 110 for performingprecomputation based on previously tracked queries. The purpose of thisroutine is to reference the query log 26 (shown in FIG. 1) for previoussearch queries 27 and to regenerate search results based on thosepreviously tracked queries 27.

Initially, previous search queries 27 are tracked using the query log 26(Block 111). Each previous search query 27 is selected as a stored query40 (Block 112) and search results based on the previous search query areregenerated (Block 113). The regenerated search results are selected asstored documents 41 (Block 114). Note that each previous search query isseparately associated with only one regenerated search result documentand multiple previous search query-regenerated search result documentassociations can be formed for any given previous search query. Iffurther previous search queries 27 remain in the query log 26 (Block115), processing continues with the next previous search query 27 (Block112). Otherwise, the routine returns.

Query Source Referencing Using Cached Data

FIG. 9 is a flow diagram showing a routine 120 for performingprecomputation based on cached documents and queries. The purpose ofthis routine is to invert pairings of cached document-cached queriesmaintained in the cache 23 for use as stored queries 40 and storeddocuments 41.

Cached documents 24 and cached queries 25 are maintained in the cache 23(shown in FIG. 1). The cached documents 24 and cached queries 25 areorganized in the cache 23 as cached query-cached document pairings.However, the associations 42 and weights 43 (shown in FIG. 3) are basedon stored document-stored query pairings. Thus, the cached query-cacheddocument pairings implicit in the organization of the cache 23 areinverted to form cached document-cached query pairings (Block 122). Aninverted search query is selected as a stored query 40 (Block 123) andthe corresponding inverted documents are selected as stored documents 41(Block 124). Note that each inverted search query is separatelyassociated with only one inverted document and multiple inverted searchquery-inverted document associations can be formed for any giveninverted search query. If further cached query-cached documents pairingsremain (Block 125), processing continues with the selection of the nextinverted pairing (Block 123). Otherwise, if no further cacheddocument-cached queries pairings remain (Block 125), the routinereturns.

On-Line Query Refinements Routine

FIG. 10 is a flow diagram showing the routine 130 for performing on-linequery refinements for use in the method 80 of FIG. 4. The purpose ofthis routine is to formulate one or more search query refinements 67preferably on-line, which can be suggested following an actual searchquery 59 issuance.

Initially, a search query 59 is issued (Block 131) and search results,in the form of search documents 60, and relevance scores 61 are received(Block 132). If possible, the stored documents 41 are matched to thesearch results (Block 133). Ideally, at least one of the search resultswill match a stored document 41. However, as the association database 39is preferably built as an off-line operation, the set of storeddocuments 41 may not fully match every possible search results.Accordingly, those search results, which do not have a matching storeddocument 41, are skipped.

Next, for each matched search result, the association 42 correspondingto the matched stored document 41 is determined and is used to retrievethe associated stored queries 40 and weights 43 (Block 134). A termvector 62 is then computed from the terms occurring in the matchedstored queries 40 and corresponding weights 43 (Block 135). Each termvector 62 is a vector in multi-dimensional space, where each dimensioncorresponds to a distinct term and each term represents an individualword or word combination. The length of a term vector 62 in eachdimension equals the sum of the weights of the corresponding term in theset of associated queries. Those term vector elements corresponding tothe terms from the original search query 59 are multiplied by a constantfactor to downwardly weight the terms to enforce independence from theoriginal search query 59. The term vectors 62 are normalized (Block136). In the described embodiment, the term vectors 62 are lengthnormalized to a length of one, although other normalizations arepossible, as would be recognized by one skilled in the art.

Clusters 63 are then formed based on the distances of the term vectors62 from a common origin (Block 137). In the described embodiment, theclusters 62 are formed using a hierarchical agglomerative clusteringalgorithm, such as described in E. Rasmussen, described supra., thedisclosure of which is incorporated by reference, although other formsof clustering could also be applied, as would be recognized by oneskilled in the art.

The resulting clusters 63 are ranked using the relevance scores 61assigned to the search documents 60 corresponding to the matched storeddocuments 41 and the number of stored documents 41 occurring in eachcluster 63 (Block 138). The highest ranking clusters are selected as thepotential refinement clusters 64 (Block 139). In the describedembodiment, the potential refinement clusters 63 are selected based on apredefined threshold value, although other cluster selection criteriaare possible, as would be recognized by one skilled in the art.

For each potential refinement cluster 64 (Block 140), a centroid 65 iscomputed (Block 141). Each centroid 65 represents the weighted center ofthe term vector 62 for each cluster 63, as a normalized sum of theproduct of the term vector 62 for each stored query 40 and the relevancescore 61 assigned to the search documents 60 corresponding to thematched stored documents 41 of the original search query 59. Otherapproaches to computing centroids 65 could also be used, including usingunweighted values and by varying the forms of weighting and averaging,as would be recognized by one skilled in the art.

A score 66 is then computed for each unique search query 59 occurring inthe potential refinement cluster 64 (Block 143). Each score 66 iscomputed as the product of the frequency of the stored query 40 for thecluster 63 times the length of the distance vector measured from theterm vector 62 of the stored query 40 to the centroid 65 of the cluster63. Other forms of scoring, ordering and ranking are possible, as wouldbe recognized by one skilled in the art. The stored query 40 with thehighest score 66 is selected as the name of the cluster 63 (Block 143).Alternatively, other cluster naming selection criteria using highest,averaged, lowest, or other forms of scoring, ordering and ranking arepossible, as would be recognized by one skilled in the art. If the score66 for the unique stored query 40 exceeds a predefined threshold (Block144), the name is added to the set of query refinements 67 (Block 145).Processing continues with each remaining potential refinement cluster 64(Block 146).

Finally, the set of refinements 67 are sorted into rankings (Block 147)as a function of the relevance scores 61 assigned to the searchdocuments 60 corresponding to the matched stored documents 41 appearingin each cluster 63 plus the size of the cluster 63 in number of storeddocuments 41. As an optional step, the set of refinements 67 areaugmented with supplemental queries (Block 148). In the describedembodiment, each supplemental query consists of the terms originallyappearing in the search query 59 and negated forms of all termsappearing in the set of refinements 67, but not appearing in theoriginal search query. Other forms of augmenting the set of refinements67 are feasible, as would be recognized by one skilled in the art. Theset of refinements 67 are presented (Block 149) based on the rankingsand, optionally, cluster scores. The routine then returns.

Integrating Candidate Query Refinements Routine

FIG. 11 is a flow diagram showing a routine 160 for integratingcandidate query refinements for use in the method 80 of FIG. 4, inaccordance with a further embodiment. The purpose of this routine is toformulate one or more search query refinements 67, which can besuggested by integrating candidate query refinements 68. The candidatequery refinements 68 can originate from any external source, as would berecognized by one skilled in the art, including the user 19 asuser-specified query refinements, third parties, and other approaches toattempting to remedy poorly-framed search queries. For brevity, thosedetailed operations previously presented above with reference to FIG. 10are described in summary fashion where applicable.

By way of example, other approaches attempt to remedy poorly-framedsearch queries by suggesting refinements, which can be considered andselected by a user as an adjunct to or in lieu of the original searchquery. One approach suggests previous, recurring search queries, whichcontain the terms used in the original search query, along with otherterms. However, the suggested queries can overlap in meaning with theoriginal search query and word co-occurrences and frequencies poorlydivide the search space into conceptually independent partitions.Another related approach tracks search query refinements entered byother users and suggests frequently-used refinements to modify theoriginal search query. However, reliance on only frequently occurringrefinements can also poorly divide the search space into conceptuallyindependent partitions. Yet another related approach clusters documentsreceived in response to an original search query based on the termsoccurring in each document. The terms that characterize each of theclusters are used as cluster names and suggested as refinements.However, the resulting refinements often consist of terms not generallyoccurring in a user-specified search query and, as a result, can bedifficult to understand and can perform poorly when used as a searchquery.

Initially, as before, a search query 59 is issued (Block 161) and searchresults, in the form of search documents 60, and relevance scores 61 arereceived (Block 162). Following search results matching and term vectorcomputation and normalization, an initial set of clusters 63 is formedbased on the distances of the term vectors 62 (Block 163). For eachcandidate query refinement 68 (Block 164), an attempt is made to assignthe candidate query refinement 68 to one of the clusters 63 (Block 165).If the candidate query refinement 68 is not assigned to a cluster 63(Block 166), the candidate query refinement 68 is added to a set oforthogonal queries (Block 167). Processing continues with each remainingcandidate query refinement 68 (Block 168).

Next, for each candidate query refinement 68 in the set of orthogonalqueries (Block 169), a search query 59 is issued (Block 170) and searchresults, in the form of search documents 60, and relevance scores 61 areagain received (Block 171). The search results are pooled into a secondset of documents (Block 172). Following search results matching and termvector computation and normalization, a second set of clusters 63 isformed based on the distances of the term vectors 62 (Block 173). Aunion is formed of the initial and second sets of clusters 63 (Block174). Following cluster ranking, potential refinement cluster 64selection, centroid 65 computation, and unique query scoring, theclusters 63 in the union are named for the unique queries with thehighest scores (Block 175). Finally, the set of refinements 67 arepresented (Block 149) based on the rankings and, optionally, clusterscores. The routine then returns.

Query Refinement Example

By way of example, a user 19 might submit a search query 59, whichincludes the individual word, “jaguar.” Upon issuance, a set of searchdocuments 60 are received and the top 100 documents are chosen forclustering. Note a set of 100 documents is used merely for the purposesof illustration and any other limit would be equally suitable, as wouldbe recognized by one skilled in the art. The search documents 60 mightnaturally identify several relevant semantic groupings, includingdocuments about automobiles manufactured by Jaguar Corporation,including hyperlink references to the official Jaguar CorporationWebsites in the United States and United Kingdom and a Jaguar brandautomobile owners association. The semantic groupings might also includedocuments about the Macintosh operating system version code-namedjaguar, documents about jaguar animals, as well as documents about anumber of other miscellaneous topics that may not be groupable intocohesive document clusters 63.

During the clustering phase, term vectors 62 are computed for each ofthe top 100 documents ranked by relevance score. As above, a set of 100documents is used merely for the purposes of illustration and any otherlimit would be equally suitable, as would be recognized by one skilledin the art. Each selected search document 60 is matched to a storeddocument 41 in the association database 39 and the corresponding storedqueries 40 are determined by looking up the associations 42 for eachmatched stored document 41. The term vectors 62 are formed by flatteningthe constituent terms for each corresponding stored query 40 into asimple vector space. Clusters 63 are generated from the term vectors 62,which typically extracts the relevant semantic groupings, such as thosegroupings described above.

A cluster centroid 65 is calculated for each cluster 63. All searchqueries 59 associated with a search document 60 in the cluster 63 arescored according to the distance from the cluster centroid 65 and thepercent of stored documents 41 occurring in the cluster 63 with whicheach stored document 41 is associated. For instance, assume that acluster 63 is calculated for the example “jaguar” search query 59 forthe semantic grouping containing documents about Jaguar brandautomobiles. In the corresponding cluster centroid 65, the dominantterms include words, such as “jaguar,” “automobile,” “auto,” “car,”“USA,” “UK,” and so forth. The best matching query name suitable as asuggested query refinement 67 would be “jaguar car,” which has goodcoverage over the entire cluster 63 and also contains the two termshaving a highest weight in the cluster centroid 65.

Similarly, further assume that a cluster 63 is calculated for thesemantic grouping containing documents about the Macintosh code-namedjaguar operating system. In the corresponding cluster centroid 65, thedominant terms include words, such as “jaguar,” “X,” “Mac,” “OS,” and soforth. The best matching query name suitable as a suggested queryrefinement 67, assuming case insensitivity, would be “mac os x jaguar,”which contains all of the top search query terms and appears in many ofthe documents in the cluster 63. Other generated clusters 63 and querynames suitable as suggested query refinement 67 include “jaguar racing”for documents about Jaguar automobile racing clubs and “jaguar cat” fordocuments about the jaguar animal.

Finally, the refinements 67 are sorted as a function of the relevancescores 61 assigned to the search documents 60 corresponding to thematched stored documents 41 appearing in each cluster 63 plus the sizeof the cluster 63 in number of stored documents 41. A cluster 63 will beranked higher than another cluster 63 if the cluster 63 is either largeror has stored documents 41 having higher relevance scores 61. In theexample, the final ranking of the refinements 67 includes “jaguar car,”“mac os x jaguar,” “jaguar racing,” and “jaguar cat.” The rankings and,optionally, cluster scores are used for presentation purposes.

In a further embodiment, the refinements 67 would include negated formsof all terms appearing in the set of refinements 67, but not appearingin the original search query. Thus, the alternative refinements 67include “jaguar -car -mac-os-x -racing -cat.” In a still furtherembodiment, a predetermined set of search queries 59 selected from pastuser queries could be used to precompute possible sets of refinements 67for the predetermined queries. The predetermined queries would be issuedand the search results would be maintained in a database for look up inresponse to user search requests based on the predetermined queries.

While the invention has been particularly shown and described asreferenced to the embodiments thereof, those skilled in the art willunderstand that the foregoing and other changes in form and detail maybe made therein without departing from the spirit and scope of theinvention.

1. A system for creating query refinement suggestions, comprising: amatcher matching at least one search document retrieved responsive to aquery to one or more stored queries; and a scorer scoring the storedquery as a potential query refinement suggestion.
 2. A system accordingto claim 1, further comprising: a document matcher matching the at leastone search document to one or more stored documents associated with theone or more stored queries.
 3. A system according to claim 1, furthercomprising: a weight associated with at least one such stored query; anda clusterer forming at least one cluster based on the stored query andweight.
 4. A system according to claim 3, further comprising: a termvector used in cluster formation computed from terms extracted from thestored query and based on the weight for the stored query.
 5. A systemaccording to claim 4, further comprising: a distance for the termvector; and the cluster forming the at least one cluster relative to thedistance.
 6. A system according to claim 3, further comprising: a rankerranking the at least one cluster relative to the at least one othercluster by evaluating a relevance score associated with each searchdocument corresponding to each matched stored document.
 7. A systemaccording to claim 6, further comprising: a selector selecting at leastone ranked cluster as a potential refinement cluster.
 8. A systemaccording to claim 3, further comprising: a centroid computed as aweighted center of the at least one cluster; and a score computed foreach stored query relative to the centroid.
 9. A system according toclaim 8, further comprising: a selector naming the at least one clusterfor at least one scored stored query.
 10. A system according to claim 1,further comprising: a threshold applied to the stored scored query. 11.A system according to claim 1, further comprising: a precomputationengine associating one or more stored documents to the stored querybased on at least one of a chosen search document, a set of searchdocuments, regenerated previous search documents, and inverted cacheddocument and query pairings.
 12. A method for creating query refinementsuggestions, comprising: matching at least one search document retrievedresponsive to a query to one or more stored queries; and scoring thestored query as a potential query refinement suggestion.
 13. A methodaccording to claim 12, further comprising: matching the at least onesearch document to one or more stored documents associated with the oneor more stored queries.
 14. A method according to claim 12, furthercomprising: associating a weight with at least one such stored query;and forming at least one cluster based on the stored query and weight.15. A method according to claim 14, further comprising: computing a termvector used in cluster formation from terms extracted from the storedquery and based on the weight for the stored query.
 16. A methodaccording to claim 15, further comprising: determining a distance forthe term vector; and forming the at least one cluster relative to thedistance.
 17. A method according to claim 14, further comprising:ranking the at least one cluster relative to the at least one othercluster by evaluating a relevance score associated with each searchdocument corresponding to each matched stored document.
 18. A methodaccording to claim 17, further comprising: selecting at least one rankedcluster as a potential refinement cluster.
 19. A method according toclaim 14, further comprising: computing a centroid as a weighted centerof the at least one cluster; and computing a score for each stored queryrelative to the centroid.
 20. A method according to claim 19, furthercomprising: naming the at least one cluster for at least one scoredstored query.
 21. A method according to claim 12, further comprising:applying a threshold to the stored scored query.
 22. A method accordingto claim 12, further comprising: associating one or more storeddocuments to the stored query based on at least one of a chosen searchdocument, a set of search documents, regenerated previous searchdocuments, and inverted cached document and query pairings.
 23. Acomputer-readable storage medium holding code for performing the methodaccording to claim
 12. 24. An apparatus for creating query refinementsuggestions, comprising: means for matching at least one search documentretrieved responsive to a query to one or more stored queries; and meansfor scoring the stored query as a potential query refinement suggestion.25. A system for providing search query refinements, comprising: anassociator associating a stored query and a stored document as a logicalpairing and assigning a weight to the logical pairing; a searcherissuing the search query and producing a set of search documents; amatcher matching at least one search document to at least one storeddocument and retrieving the stored query and the assigned weightassociated with the matching at least one stored document; a clustererforming at least one cluster based on the stored query and the assignedweight associated with the matching at least one stored document; and ascorer scoring the stored query associated with the matching at leastone stored document for the at least one cluster relative to at leastone other cluster and suggesting at least one such scored search queryas a set of query refinements.
 26. A system according to claim 25,further comprising: a selector selecting one such search document chosenfrom among the set of search documents responsive to the search queryissuance as the at least one such search document.
 27. A systemaccording to claim 25, further comprising: a selector selecting the setof search documents as the at least one such search document.
 28. Asystem according to claim 25, further comprising: a query log trackingprevious search queries; and a regenerator regenerating a set ofprevious search documents produced by the previous search queries as theat least one such search document.
 29. A system according to claim 25,further comprising: a cache associating at least one cached document andone or more cached queries as a cached pairing; and an inverterinverting each cached pairing to associate at least one cached query andone or more cached documents as the at least one such search document.30. A system according to claim 25, wherein relevancy to the storedquery is estimated for the stored document as the weight assigned to thepairing.
 31. A system according to claim 30, wherein each such assignedweight for a plurality of pairings corresponding to the stored query andthe stored document is summed.
 32. A system according to claim 25,wherein each stored query comprises one or more terms, furthercomprising: a term vector comprising the terms in the stored queryassociated with the matching at least one stored document; a distancedetermined for the term vector; and the clusterer forming the at leastone cluster relative to the distance.
 33. A system according to claim32, further comprising: a normalizer normalizing the term vector.
 34. Asystem according to claim 32, further comprising: an evaluator computinga length of the term vector in multi-dimensional space with eachdimension equaling a sum of the weights of the term in a set ofassociated stored queries.
 35. A system according to claim 32, furthercomprising: a relevance score assigned to the at least one searchdocument; and a ranker ranking the at least one cluster relative to theat least one other cluster by the relevance score associated with thematching at least one search document and a number of the matching atleast one search document.
 36. A system according to claim 35, furthercomprising: a selector selecting one of more of the ranked at least onecluster as potential refinement clusters based on the rankings.
 37. Asystem according to claim 36, further comprising: a centroid computed asa weighted center for each such potential refinement cluster; and thescorer scoring the stored query associated with the matching at leastone stored document for the potential refinement cluster relative to thecentroid.
 38. A system according to claim 37, further comprising: anevaluator computing the centroid as a normalized sum of a product of theterm vector for each stored query and the relevance score associatedwith the matching at least one search document.
 39. A system accordingto claim 38, further comprising: a length of a distance vectordetermined from the term vector and the centroid; and the scorercomputing the score for the scored query as a product of a number ofstored documents with which the stored query is associated and thedistance vector length.
 40. A system according to claim 25, furthercomprising: a selector selecting the stored query associated with thematching at least one stored document relative to a threshold.
 41. Asystem according to claim 25, further comprising: a sorter sorting theset of query refinements.
 42. A system according to claim 25, furthercomprising: a presenter presenting the set of query refinements.
 43. Asystem according to claim 25, further comprising: a set of supplementalquery refinements negating each term in the set of query refinements notpresent in the search query and using the negated terms in combinationwith the search query as at least one supplemental query refinement. 44.A system according to claim 25, further comprising: an associationdatabase maintaining the pairings.
 45. A system according to claim 25,wherein at least one of each such stored document and each such searchdocument is specified as at least one of a Uniform Resource Locator(URL), hyperlink, anchor, and document excerpt.
 46. A method forproviding search query refinements, comprising: associating a storedquery and a stored document as a logical pairing and assigning a weightto the logical pairing; issuing the search query and producing a set ofsearch documents; matching at least one search document to at least onestored document and retrieving the stored query and the assigned weightassociated with the matching at least one stored document; forming atleast one cluster based on the stored query and the assigned weightassociated with the matching at least one stored document; and scoringthe stored query associated with the matching at least one storeddocument for the at least one cluster relative to at least one othercluster and suggesting at least one such scored search query as a set ofquery refinements.
 47. A method according to claim 46, furthercomprising: selecting one such search document chosen from among the setof search documents responsive to the search query issuance as the atleast one such search document.
 48. A method according to claim 46,further comprising: selecting the set of search documents as the atleast one such search document.
 49. A method according to claim 46,further comprising: tracking previous search queries; and regenerating aset of previous search documents produced by the previous search queriesas the at least one such search document.
 50. A method according toclaim 46, further comprising: associating at least one cached documentand one or more cached queries as a cached pairing; and inverting eachcached pairing to associate at least one cached query and one or morecached documents as the at least one such search document.
 51. A methodaccording to claim 46, further comprising: for each such pairing,estimating relevancy to the stored query for the stored document as theweight assigned to the pairing.
 52. A method according to claim 51,further comprising: summing each such assigned weight for a plurality ofpairings corresponding to the stored query and the stored document. 53.A method according to claim 46, wherein each stored query comprises oneor more terms, further comprising: computing a term vector comprisingthe terms in the stored query associated with the matching at least onestored document; determining a distance determined for the term vector;and forming the at least one cluster relative to the distance.
 54. Amethod according to claim 53, further comprising: normalizing the termvector.
 55. A method according to claim 53, further comprising:computing a length of the term vector in multi-dimensional space witheach dimension equaling a sum of the weights of the term in a set ofassociated stored queries.
 56. A method according to claim 53, furthercomprising: assigning a relevance score to the at least one searchdocument; and ranking the at least one cluster relative to the at leastone other cluster by the relevance score associated with the matching atleast one search document and a number of the matching at least onesearch document.
 57. A method according to claim 56, further comprising:selecting one of more of the ranked at least one cluster as potentialrefinement clusters based on the rankings.
 58. A method according toclaim 57, further comprising: computing a centroid as a weighted centerfor each such potential refinement cluster; and scoring the stored queryassociated with the matching at least one stored document for thepotential refinement cluster relative to the centroid.
 59. A methodaccording to claim 58, further comprising: computing the centroid as anormalized sum of a product of the term vector for each stored query andthe relevance score associated with the matching at least one searchdocument.
 60. A method according to claim 59, further comprising:determining a length of a distance vector from the term vector and thecentroid; and computing the score for the scored query as a product of anumber of stored documents with which the stored query is associated andthe distance vector length.
 61. A method according to claim 46, furthercomprising: selecting the stored query associated with the matching atleast one stored document relative to a threshold.
 62. A methodaccording to claim 46, further comprising: sorting the set of queryrefinements.
 63. A method according to claim 46, further comprising:presenting the set of query refinements.
 64. A method according to claim46, further comprising: negating each term in the set of queryrefinements not present in the search query and using the negated termsin combination with the search query as at least one supplemental queryrefinement.
 65. A method according to claim 46, further comprising:maintaining the pairings in a database.
 66. A method according to claim46, further comprising: specifying at least one of each such storeddocument and each such search document as at least one of a UniformResource Locator (URL), hyperlink, anchor, and document excerpt.
 67. Acomputer-readable storage medium holding code for performing the methodaccording to claim
 46. 68. An apparatus for providing search queryrefinements, comprising: means for associating a stored query and astored document as a logical pairing and means for assigning a weight tothe logical pairing; means for issuing the search query and means forproducing a set of search documents; means for matching at least onesearch document to at least one stored document and means for retrievingthe stored query and the assigned weight associated with the matching atleast one stored document; means for forming at least one cluster basedon the stored query and the assigned weight associated with the matchingat least one stored document; and means for scoring the stored queryassociated with the matching at least one stored document for the atleast one cluster relative to at least one other cluster and means forsuggesting at least one such scored search query as a set of queryrefinements.
 69. A system for integrating query refinement candidates,comprising: a matcher matching at least one search document retrievedresponsive to a query to one or more stored documents associated with astored query and weight and matching at least one further searchdocument retrieved responsive to a candidate query to the one or morestored documents; a cluster forming at least one cluster based on thestored query and weight associated with each stored document matchedresponsive to the query and forming at least one further cluster basedon the stored query and weight associated with each stored documentmatched responsive to the candidate query; a combiner combining the atleast one cluster and the at least one further cluster; and a scorerscoring the stored query for the combined cluster relative to at leastone other cluster as a potential query refinement suggestion.
 70. Asystem according to claim 69, further comprising: a set of candidatequery refinements comprising at least one such candidate query.
 71. Asystem according to claim 70, further comprising: an evaluator assigningat least one such candidate query to the at least one cluster.
 72. Asystem according to claim 71, further comprising: a builder creating anorthogonal set of candidate query refinements comprising at least onesuch unassigned query candidate.
 73. A method for integrating queryrefinement candidates, comprising: matching at least one search documentretrieved responsive to a query to one or more stored documentsassociated with a stored query and weight; forming at least one clusterbased on the stored query and weight associated with each storeddocument matched responsive to the query; matching at least one furthersearch document retrieved responsive to a candidate query to the one ormore stored documents; forming at least one further cluster based on thestored query and weight associated with each stored document matchedresponsive to the candidate query; combining the at least one clusterand the at least one further cluster; and scoring the stored query forthe combined cluster relative to at least one other cluster as apotential query refinement suggestion.
 74. A method according to claim73, further comprising: assembling a set of candidate query refinementscomprising at least one such candidate query.
 75. A method according toclaim 74, further comprising: assigning at least one such candidatequery to the at least one cluster.
 76. A method according to claim 75,further comprising: creating an orthogonal set of candidate queryrefinements comprising at least one such unassigned query candidate. 77.A computer-readable storage medium holding code for performing themethod according to claim
 73. 78. An apparatus for integrating queryrefinement candidates, comprising: means for matching at least onesearch document retrieved responsive to a query to one or more storeddocuments associated with a stored query and weight; means for formingat least one cluster based on the stored query and weight associatedwith each stored document matched responsive to the query; means formatching at least one further search document retrieved responsive to acandidate query to the one or more stored documents; means for formingat least one further cluster based on the stored query and weightassociated with each stored document matched responsive to the candidatequery; means for combining the at least one cluster and the at least onefurther cluster; and means for scoring the stored query for the combinedcluster relative to at least one other cluster as a potential queryrefinement suggestion.